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Abstract. This article presents the results of the Model Checking Contest held at Petri Nets 2012 
in Hambourg. This contest aimed at a fair and experimental evaluation of the performances of 
model checking techniques applied to Petri nets. This is the second edition after a successful one 
in 2011 [29]. 

The participating tools were compared on several examinations (state space generation and evalua- 
tion of several types of formula? - structural, reachability, LTL, CTL) run on a set of common models 
(Place/Transition and Symmetric Petri nets). 

After a short overview of the contest, this paper provides the raw results from the context, model per 
model and examination per examination. 

Keywords: Petri Nets, Model Checking, Contest. 



1 Introduction 

When verifying by model checking a system with formal methods, such as Petri nets, one may have 
several questions such as: 

"When creating the model of a system, should we use structural analysis or an explicit model 
checker to debug the model?" 

"When verifying the final model of a highly concurrent system, should we use a symmetry- 
based or a partial order reduction-based model checker?" 

"When updating a model with large variable domains, should we use a decision diagram- 
based or an abstraction-based model checker?" 
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Results that help to answer these questions are spread among numerous papers in numerous con- 
ferences. The choice of the models and tools used in benchmarks is rarely sufficient to answer these 
questions. Benchmark results are available a long time after their publication, even if the computer ar- 
chitecture has changed a lot. Moreover, as they are executed over several platforms and composed of 
different models, conclusions are not easy. 

The objective of the Model Checking Contest © Petri nets is to compare the efficiency of verification 
techniques according to the characteristics of the models. To do so, the Model Checking Contest com- 
pares tools on several classes of models with scaling capabilities, e.g., values that set up the "size" of the 
associated state space. 

Through a benchmark, our goal is to identify the techniques that can tackle a given type of problem 
identified in a "typical model", for a given class of problem (e.g., state space generation, evaluation of 
reachability or temporal formulaae, etc.). 

The second edition of the Model Checking Contest @ Petri nets took place within the context of 
the Petri Nets and ACSD 2012 conferences, in Hamburg, Germany 7 . The original submission procedure 
was published early March 2012 and submissions gathered by mid-May 2012. After some tuning of the 
execution environment, the evaluation procedure was operated on a cluster early June. Results were 
presented during the SUMo workshop, on June 26th, 2012. 

The goal of this paper is to report the raw data provided by this second edition of the Model Checking 
Contest. It reflects the vision of MCC'2012 organizers, as it was first presented in Hamburg. All tool 
developers are listed in Section 9. 

The article is structured as follows. Section 2 presents the models proposed in this second edition. 
Then, Section 3 lists some information on the participating tools. Section 2 provides an overview of the 
evaluation methodology. Finally, Sections 5 to 7 present the raw data we collected from MCC'2012. 



7 A First edition took place in Newcastle, UK, withing the context of the SUMo workshop, associated to the Petri 
Nets et ACSD 2011 conferences [29]. 
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2 Selected Models 

A first innovation in MCC'2012 was to add a "call for model". The idea was to gather benchmarks 
from the community. Then, on top of the 7 models proposed in the first edition, we got 12 models 
from various institutes, usually coming from some tool benchmarks. When models were colored, we 
requested submitters to provide both the colored version and the Place/Transition (P/T) equivalent ver- 
sion. 

The models were provided in PNML format [26,27]. Tools developers could use the PNML descrip- 
tion to build the one of their tool. PNML was then used as the reference. 17 out of the 19 models had 
scaling parameters. Thus, we could process them with various values and then see how tools could scale 
up with these models. 

We provide below a brief description of models. Moreover, each one has a data-sheet available on 
MCC'2012 web site: http : //mcc . Iip6 . f r/2012. 

2.1 Models from MCC '2011 

These first seven models are from the MCC'2011. In this set, MAPK is the only model coming from 
a case study (biology). 

FMS belongs to the GreatSPN and SMART [12] benchmarks. It models a Flexible Manufacturing Sys- 
tem [11]. The scaling parameter corresponds to the number of initial tokens held in three places. The 
following values were used: 2, 5, 10, 20, 50, 100, 200, 500. 

Kanban [10] models a Kanban system. The scaling parameter corresponds to the number of initial 
tokens held in four places. The following values were used: 5, 10, 20, 50, 100, 200, 500, 1 000. 

MAPK models a biological system: the Mitogen-Activated Protein Kinase Kascade [23]. The scaling 
parameter changes the initial number of tokens held in seven places. The following values were used: 
8,20,40,80,160,320. 

Peterson models Peterson's mutual exclusion algorithm [35] in its generalized version for N processes. 
This algorithm is based on shared memory communication and uses a loop with N - 1 iterations, each 
iteration is in charge of stopping one of the competing processes. The scaling parameter is the number 
of involved processes. The following values were used: 2,3,4, 5, 6. 

Philosophers models the famous Dining Philosophers problem introduced by E.W Dijkstrain 1965 [46] 
to illustrate an inappropriate use of shared resources, thus generating deadlocks or starvation. The scal- 
ing parameter is the number of philosophers. The following values were used: 5, 10, 20, 50, 100, 500, 
1000, 5000, 10000, 50000, 100000. 

SharedMemory is a model taken from the GreatSPN benchmarks [8]. It models a system composed 
of P processors competing for the access to a shared memory (built with their local memory) using a 
unique shared bus. The scaling parameter is the number of processors. The following values were used: 
5, 10, 20, 50, 100, 200, 500, 1 000, 2 000, 5 000, 10 000, 20 000, 50 000. 

TokenRing is another problem proposed by E.W. Dijkstra [18]. It models a system where a set of ma- 
chines is placed in a ring, numbered to N- 1. Each machine i only knows its own state and the state 
of its left neighbor, i.e., machine (i - 1) mod (AT). Machine number plays a special role, and it is called 
the "bottom machine". A protocol ensuring non-starvation determines which machine has a "privilege" 
(e.g. the right to access a resource). The scaling parameter is the number of machines. The following 
values were used: 5, 10,15, 20, 30, 40, 50, 100, 200, 300, 400, 500. 

2.2 New Models in MCC'2012 

These twelve models were submitted by the community for MCC'2012. In this new set, several mod- 
els are coming from larger case studies: neo-election, planning, and ring. 
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cs repetitions models a client/ server application with C clients and S servers. Communication from 
clients to servers is not reliable, with requests stored in a buffer of size B. Communication from servers 
to clients are reliable. A client send its message until it receives an answer. The scaling parameter is a 
function of C for a fixed number of severs. The following values were used: 25,49, 100,225,400,625,900. 

echo This file specifies the Echo Algorithm (see [38]) for grid like networks. It is a protocol for propaga- 
tion of information with feedback in a network. A distinguished agent (initiator), starts the distribution 
of a message by sending it to all its neighbors. On receiving some first message, every other agent for- 
wards the message to all its neighbors, except the one it received its first message from. Then it awaits 
messages from all recipients of its forwards (regardless whether these messages had been intended as 
forwards or acknowledgments) and replies to the agent where it received its first message from. As soon 
as the initiator receives a message from all its neighbors, the protocol terminates. In this example, agents 
are arranged in a hypercube that can be scaled in two values: D, the number of dimensions and R, the 
number of agents per dimensions. The scaling parameter is a combination of D and R. The following 
values were used: d2r\\, d2r\3, d2rl5, d2r\l , d2rl9, d2r9, d3r3, d3r5, d3rl, d4r3, d5r3. 

eratosthenes This model implements the sieve of Eratosthenes [47] . The scaling parameter is the size 
of the sieve. The following values were used: 5, 10, 20, 50, 100, 200, 500. 

galloc res It models the deadlock-free management of mutually exclusive resources known as the 
"global allocation strategy" [31]. When a process enters a critical section, it locks all the resources needed 
to be used in the critical section (in the model, 4 max). Then, it can release a subset of these resources 
(max 2 in the model) at a time (and then stay in the critical section) or exit the critical section, thus 
releasing all the remaining resources it locks. The scaling parameter is a value N for N processes and 
N x2 resources. The following values were used: 3,5,6,7,9, 10, 11. 

la m port fmea It models Lamport's fast mutual exclusion algorithm designed for multi-processor ar- 
chitectures with a shared memory and was studied in [28] . The scaling parameter is the number of pro- 
cesses competing for the critical section. The following values were used: 2, 3, 4, 5, 6, 7, 8. 

neo-election The Neo protocol aims at managing large distributed databases on clusters of worksta- 
tions. The machines on the cluster may have several roles. This model focusses on master nodes which 
handle the communications between all nodes, and in particular requests for accessing database ob- 
jects. Prior to that all master nodes agree on a primary master which will be the operating one, the other 
master nodes being secondary, waiting to replace the primary master if needed. This model specifies 
this election algorithm [9] . The scaling parameter is the number of master nodes. The following values 
were used: 2,3,4,5,6,7,8. 

philo dyn is a variation of the Dining Philosophers where philosophers can join or quit the table [5] . 
Each philosopher has its own fork, as in the usual version. The interesting point is that identifiers of 
left and right for each philosopher must be computed or stored somewhere. A philosopher can enter 
the table only if the two forks around his position are available. He can leave if his fork is free, and he 
is thinking. The scaling parameter is the maximum number of philosophers. The following values were 
used: 2,3,10,20,50,80. 

planning It models the equipment (displays, canvases, documents, and lamps) of a smart conference 
room of the University of Rostock. It was derived from a proprietary description format that was used by 
an AI planning tool to generated plans to bring the room in a desired state, for instance displaying a doc- 
ument on a certain canvas while switching off the lights. This problem can be expressed as a reachability 
problem. This model has no scaling parameter. 

railroad it corresponds to the Petri nets semantics of an ABCD model of a railroad crossing system, t 
has three components: a gate sub-net, a controller sub-net and n tracks sub-nets that differ only by an 
identifier k in {0, . . ., n - 1}. These components communicate through shared places, some being low- 
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level places to exchange signals, others being integer-valued places to exchange tracks identifiers. The 
controller also has a place to count the number of trains at a given time. The scaling parameter is the 
number of tracks. The following values were used: 5, 10, 20, 50, 100. 

ring It models a three-module ring architecture [17]. The communication architecture contains as 
many channels as there are modules. It tests the occurrence of global deadlock arising from a local one. 
It uses stoppable clocking scheme on arbitrated input and output channels. This model has no scaling 
parameter. 

rwmutex It models a system with readers and writers [38]. Reading can be conducted concurrently 
whereas writing has to be done exclusively. This is modeled by a number of semaphores (one for each 
reader) that need to be collected by a writer prior to writing. The scaling parameter a combination of R, 
the number of readers and W, the number of writers. The following values were used: r 10 if 10, rl0w20, 
rl0w50, rlOwlOO, rl0w500, rlOwlOOO, rl0w2000, r20wl0, rlOOw/10, r500wl0, rlOOOwlO, r2000w/10. 

simple lbs models a simple load balancing system composed of a set of clients, two servers, and be- 
tween these, a load balancer process. The scaling parameter is the number of clients to be balanced over 
the servers. The following values were used: 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20. 
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3 Participating Tools 

We got 10 tool submissions summarized in Table 1: AlPiNA, Crocodile, Helena, ITS-Tools, LoLa 
-binstore, LoLa-bloom, Marcie, Neco, PNXDD, and Sara. Their descriptions, written by the tool de- 
velopers, are given in this section. 

AlPiNA 8 [25] stands for Algebraic Petri nets Analyzer and is a symbolic model checker for Algebraic 
Petri nets. It can verify various state properties expressed in a first order logic property language. 

Algebraic Petri nets(APNs) (Petri nets + Abstract Algebraic Data Types) is a powerful formalism to 
model concurrent systems in a compact way. Usually, concurrent systems have very large sets of states, 
that grow very fast in relation to the system size. Symbolic Model Checking (DD -based one) is a proven 
technique to handle the State Space Explosion for simpler formalisms such as Place/Transition nets. 
AlPiNA extend these concepts to handle algebraic values that can be located in net places. 

For this purpose AlPiNA uses enhanced DDs such as Data Decision Diagrams and Set Decision Di- 
agrams for representing the place vectors and Z Decision Diagrams [3] for representing the values of the 
APN. It also allows to specify both algebraic and topological clusters to group states together and thus 
to reduce the memory footprint. Particular care has been taken to let users freely model their systems in 
APNs and in a second independent step to tune the optimization parameters such as unfolding rules, 
variable order, and algebraic clustering. Compared to Colored net approaches, AlPiNA [4] solves prob- 
lems related to the unbounded nature of data types and uses the inductive nature of Abstract Algebraic 
Data Types to generalize the unfolding and clustering techniques to any kind of data structure. 

AlPiNA's additional goal is to provide a user friendly suite of tools for checking models based on 
the Algebraic Petri nets formalism. In order to provide great user experience, it leverages the Eclipse 
platform. 

Crocodile 9 [15] was initially designed as a demonstration tool for the so-called symbolic/symbolic 
approach [43] . It combines two techniques for handling the combinatorial explosion of the state space 
that are both called "symbolic". 

The first "symbolic" technique concerns the reduction of the reachability graph of a system by its 
symmetries. The method used in C rocodile was first introduced in [6] for the Symmetric nets, and was 
then extended to the Symmetric nets with Bags (SN B) in [21] . A symbolic reachability graph (also called 
quotient graph) can be built for such types of Petri nets, thus dramatically reducing the size of the state 
space. 

The second "symbolic" technique consists in storing the reachability graph using decision diagrams, 
leading to a symbolic memory encoding. Crocodile relies on Hierarchical Set Decision Diagrams [16]. 
These present several interesting features, such as hierarchy, and the ability to define inductive opera- 
tions. 



Tool is available at http : / / alpina . unige . ch. 

Tool is available at http : //move . Iip6 . f r/ sof tware/Crocodile. 



Tool Name 


Team 


Institution 


Country 


Contact Name 


AlPiNA 


CUI/SMV 


Univ. Geneva 


Switzerland 


D. Buchs 


Crocodile 


LIP6/MoVe 


UPMC 


France 


M. Colange 


Helena 


LIPN/LCR 


Univ. Paris 13 


France 


S. Evangelista 


ITS-Tools 


LIP6/MoVe 


UPMC 


France 


Y. Thierry-Mieg 


LoLA 


Team Rostock 


Univ. Rostock 


Germany 


N. Lohmann & K. Wolf 


Marcie 


DSSZ 


Univ. Cottbus 


Germany 


M. Heiner & C. Rohr 


Neco 


IBISC 


Univ. Evry 


France 


L. Fronc 


PNXDD 


LIP6/MoVe 


UPMC 


France 


E. Paviot-Adet 


Sara 


Team Rostock 


Univ. Rostock 


Germany 


H. Wimmel & K. Wolf 



Table 1. Summary of data on participating tools 
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Still under development, Crocodile essentially generates the state space of a SNB and then pro- 
cesses reachability properties. The version submitted for this second edition of the Model Checking 
Contest includes new strategies for the management of bags [14]. 

Helena 10 [19] is an explicit state model checker for High Level Petri nets. It is a command-line tool 
available available under the GNU GPL. 

Helena tackles the state explosion problem mostly through a reduction of parallelism operated at 
two stages of the verification process. First, static reduction rules are applied on the model in order 
to produce a smaller net that - provided some structural conditions are verified - is equivalent to the 
original one but has a smaller reachability graph. Second, during the search, partial order reduction is 
employed to limit, as much as possible, the exploration of redundant paths in the reachability graph. 
This reduction is based on the detection of independent transitions of the net at a given marking. Other 
reduction techniques are also implemented by Helena, e.g., state compression, but were disabled dur- 
ing the contest due to their inadequacy with the proposed models. 

ITS-Tools 11 [44] are a set of tools to analyze Instantiate Transition Systems, introduced in [44]. This 
formalism allows compositional specification using a notion of type and instance inspired by compo- 
nent oriented models. The basic elementary types are labeled automata structures, or labeled Petri nets 
with some classical extensions (inhibitor arcs, reset arcs...). The instances are composed using event- 
based label synchronization. 

The main strength of ITS-Tools is that they rely on Hierarchical Set Decision Diagrams [16] to 
perform analysis. These decision diagrams support hierarchy, allowing to share representation of states 
for some subsystems. When the system is very regular or symmetric, recursive encodings [44] may even 
allow to reach logarithmic overall complexity when performing analysis. Within the contest, the Philoso- 
phers and TokenRing examples proved to be tractable using this recursive folding feature. 

Set Decision Diagrams also offer support for automatically enabling the "saturation" algorithm for 
symbolic least fixpoint computations [22], a feature allowing to drastically reduce time and memory 
consumption. This feature was used in all computations. 

LoLA 12 [49] is an explicit Petri net state space verification tools. It can verify a variety of properties 
ranging from questions regarding single Petri net nodes (e.g., boundedness of a place or quasiliveness 
of a transition), reachability of a given state or a state predicate, typical questions related to a whole 
Petri net (e.g., deadlock freedom, reversibility, or boundedness), and the validity of temporal logical 
formulae such as CTL. It has been successfully used in case studies from various domains, including 
asynchronous circuits, biochemical reaction chains, services, business processes, and parameterized 
Boolean programs. 

For each property, LoLA provides tailored versions of state space reduction techniques such as stub- 
born sets, symmetry reduction, coverability graph generation, or methods involving the Petri net invari- 
ant calculus. Depending on the property to be preserved, these techniques can also be used in combi- 
nation to only generate a small portion of the state space. 

Two versions of LoLA were submitted in 2012. LoLa-binsto re is a complete reimplementation of 
the tool since 2011. LoLa-bloom uses a bit hashing technique. 

Marcie 13 [41] is a tool for the analysis of Generalized Stochastic Petri Nets, supporting qualitative and 
quantitative analyses including model checking facilities. Particular features are symbolic state space 
analysis including efficient saturation-based state space generation, evaluation of standard Petri net 
properties as well as CTL model checking. 

Most of Marcie's features are realized on top of an Interval Decision Diagram implementation [45]. 
IDDs are used to efficiently encode interval logic functions representing marking sets of bounded Petri 

10 Tool is available at http : / /helena-mc . sourcef orge .net. 

11 Tool is available at http : //ddd . Iip6 . f r. 

12 Tool is available at http : / /www. informatik.uni-rostock.de/tpp/lola. 

13 Tool is available at http : //www-dssz . informatik.tu-cottbus.de/DSSZ/Software/Marcie. 
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nets. This allows to efficiently support qualitative state space based analysis techniques [42] . Further, 
Ma rcie applies heuristics for the computation of static variable orders to achieve small decision diagram 
representations. 

For quantitative analysis Ma rcie implements a multi-threaded on-the-fly computation of the un- 
derlying CTMC. It is thus less sensitive to the number of distinct rate values than approaches based on, 
e.g., Multi-Terminal Decision Diagrams. Further it offers symbolic CSRL model checking and permits 
to compute reward expectations. Additionally Ma rcie provides simulative and explicit approximative 
numerical analysis techniques. 

Neco 14 is a coloured Petri nets compiler which produces libraries for explicit model-checking. These 
libraries can be used to build state spaces. It is a command-line tool available under the GNU Lesser 
GPL. 

Neco is based on the SNAKES [36] toolkit and handles coloured Petri nets annotated with arbitrary 
Python [37] objects. This allows for a high degree of expressivity. Extracting information from models, 
Neco can identify object types and produce optimized Python or C exploration code. The later is done 
using the Cython [2] language. Moreover, if a part of the model cannot be compiled efficiently a Python 
fallback is used to handle this part of the model. 

Neco also performs model based optimizations using place bounds [20] and control flow places for 
more efficient firing functions. However, these optimizations are closely related to a modelling language 
we use which allows them to be assumed by construction. Because the models from the contest were 
not provided with such properties, these optimizations could not be used. 

PNXDD 15 generates the state-space of Place/Transition nets. When Colored nets are used in the Model 
Checking Contest, equivalent P/Ts are obtained after an "optimized" unfolding [30] (unused places and 
transitions are detected and suppressed). 

State space storage relies on Hierarchical Set Decision Diagrams [16] (SDDs). These are decision 
diagrams with any data type associated to arcs (see e.g., [33] for an overview of DD-like structures). If the 
associated data type is another SDD, hierarchical structures can be constructed. 

Since PNXDD exploits hierarchy, a state is seen as a tree, where the leaves correspond to places mark- 
ing. This particular structure offers greater sharing opportunities than a, for instance, vector-based rep- 
resentation. The conception of such a tree is critical to reach good performances and heuristics are 
being elaborated for this purpose [24]. The one used for the Model Checking Contest is based on [1]: for 
colored models that do scale via the size of color types, PNXDD uses a tree-like version of this heuristic, 
while the original version is kept when colored models only scale via the number of tokens in the initial 
marking. 

Sara 16 uses the state equation, known to be a necessary criterion for reachability and in a modified 
way also for other properties like coverability, to avoid enumerating all possible states of a system [39] . A 
minimal solution of the state equation in form of a transition vector is transformed into a tree of possible 
firing sequences for this solution. A firing sequence using all the transitions given in the solution (with 
the correct multiplicity) reaches the goal. 

For tree pruning, partial order reduction is used, especially in the form of stubborn sets [40,32] . If the 
goal cannot be reached using the obtained solution, places that do not get enough tokens are computed. 
Constraints are built and added to the state equation (in a CEGAR-like fashion [13]). These constraints 
modify the former solution by adding transition invariants, temporarily allowing for additional tokens 
on the undermarked places. 

Sara detects unreachable goals either from an unsatisfiable state equation or by cutting the solution 
space to finite size when the repeated addition of transition invariants is known not to move towards the 
goal. A more involved explanation of the algorithm behind Sa ra can be found in [48]. 



Tool is available at http : / / code . google . com/p/neco-net- compiler/. 

5 Tool is available at https : //srcdev . Iip6 . fr/trac/research/NEOPPOD/wiki/pnxdd. 

6 Tool is available at http : / /www. service - 1 echnology . org/tools/download. 
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4 Evaluation Methodology 

Roughly, the evaluation methodology was the same as for MCC'2011 (it is presented in [29]). The 
main differences were the following: 

1. we changed some of the examinations, to allow checking of more properties; 

2. we monitored tools by means of virtual machines, which is more complex to operate but much less 
intrusive, thus leading to more accurate results. 

4.1 Examinations 

There were two types of examinations in MCC'2011: state space generation and evaluation of reach- 
ability formulae. We enriched this second examination by dividing it with subclasses of formulas: 

- Structural: these only refer to structural aspects of the net such as existence and value of a place 
bound, level of transition liveness, and the existence of a deadlock; 

- Reachability: these only refer to a combination of profile over states by comparing token values in 
places and/ or computing transition fireability; 

- LTL: these formulas contain LTL operators mixed with reachability ones; 

- CTL: these contain CTL operators mixed with reachability ones. 

All details on the formulas and the associated language is described in [34] . 

We decided to operate a set of formulas per model and per scaling parameter when it existed. Since 
models were proposed in both colored and P/T versions, we also proposed the formulas in both types. 

These two choices made the preparation of the contest quite difficult. First, computing equivalent 
formulas for colored nets and their P/T equivalent is sometimes not possible. Second, we had no time to 
check that formulas could be satisfied or unsatisfied as for the MCC'2012. This impede the results of the 
formulas examinations as we state in Section 9. 

4.2 Use of Virtual Machines 

Tools were submitted within a disk image that was provided with a Linux distribution (Windows 
could have been proposed but no tool requested it). 

Then, we processed the examination on a 23-nodes cluster equipped with PowerEdge R410 (6 ports 
gigabits) and 1.5To local disks. Each node operated an Intel Xeon E5645@2.40GHz (6 cores, 12 threads) 
and had 8GB of memory (DDR3, 1333). 

We considered one examination on one model for a given scaling parameter (when there was one) 
as a run. The contest required 2419 run in total (639 for the state space examination and 1780 for the 
structural and reachability examinations). This does not include the necessary trials required to evaluate 
the procedure and check (with their developers) that tools were operating appropriately. 

Input: M, a set of scalable models to be processed 
foreach me M do 

launch the virtual machine m 
foreach v, scaling values for m do 

foreach e, in StateSpace, StrctFormulae, ReachFormulae, CTLFormulae, LTLFormulae do 
connect to the virtual machine 
request e on m for value v 

if Virtual machine reports confinement error then 
L continue to next model 

Operate an epilogue for m 

Algorithm 1: Actions performed for each tool by the invocation script 

The "examinations" requested for the contest were performed thanks to an invocation script that 
iterated invocation of each tool over models instances. This invocation script, adapted from the one of 
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the first edition, is presented in Algorithm 1. The major improvement is to stop executing the tool as 
soon as one instance fails. Another concern (that makes the script more complex without changing its 
principle) was to increase the use of parallelism over the nodes of the cluster. 
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5 Raw Data for the State Space Generation 




FMS 
Kanban 
MAPK 
Peterson 
Philosophers 
Shared-Memory 
TokenRing 



Cs repetitions 

Echo 
Eratosthenes 

Galloc res 
Lamport fmea 
NEO-election 
Philo dyn 
Planning 
Railroad 

Ring 
Rw mutex 
Simple lbs 



Table 2. Results for the state space generation examination 



This section shows the raw results of the state space generation examination. Table 2 summarizes 
the highest scaling parameter reached by the tools for each model. Then Charts generated from the data 
collected for this examination are provided. 

This table, as well as Tables 4 and 6, should be interpreted using the legend below (a value shows the 
maximum scaling parameter and, on the rightmost part of the cell, the type of Petri net used by the tool 
is indicated - P for P/T nets or C for Colored ones): 

UThe tool does not participate. 
The tool participates, but cannot compute for any scaling value. 
The tool participates. 
The tool participates and reaches the best parameter among tools. 
The tool participates and reaches the maximum parameter. 

Section 5.1 presents the data computed by tools. Section 5.2 shows how models have been handled 
by tools and Sections 5.3 and 5.4 summarize how tools did cope with models. 

Then, Section 5.5 presents the charts showing the evolution of memory and CPU according to the 
scaling parameter (if any). Only models handled by at least one tool are displayed. 

Finally, Sections 5.6 to 5.12 show the evolution of memory and CPU consumption while tools are 
performing the state space generation for a model. 

No comment is provided intentionally, the objective of this document mainly being to report all the 
data we collected this year. 

5. 1 Computed sizes for the State Spaces 

This section presents the data computed by tools for the various state spaces that were proposed. 
This information is summarized in Table 3: columns "scale" shows the value of the scaling parameter 
and the column "|S|" aside shows the state space size corresponding to this value. "?" are displayed when 
the value is unknown (i.e. no tool could compute the value). In one situation, we display oo since it was 
reported that the big numbers C++ library fails to compute the value. 
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Table 3. Size of computed state spaces 
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In some cases, the value returned by tools differ. For Crocodile, this is normal because it counts 
symbolic states (i.e. the ones of the quotient graph) in the sense of [7] while other tools count explicit 
states. 

This also concerns other tools but it can be due to the use of specific techniques preventing the 
computation of some "useless states" like the use of partial order techniques. So we only provide the 
values when tools agree or when only one tool reaches a value, when the values that are also computed 
by some other tools are confirmed. 

We thus have three "unsafe" situations remaining where we prefer not to display any value: 

1. philo dyn: for the two first values of the scaling parameter, PNXDD, Ma rcie and ITS-Tools agree 
on the size of the state space (AlPiNA and Helena do not) ; but, for higher values, only Helena is 
able to compute the state space. 

2. ring: here, only ITS-Tools and Ma rcie are able to cope with this model but they disagree. 

3. SharedMemory: for the values > 50, only Crocodile and Ma rcie are able to build the state space. 
However, Ma rcie disagree with other tools on the first results and Crocodile does not count the 
same type of states. 

5.2 Processed Models 

This section summarizes how models were processed by tools. Let us first note that no tool suc- 
ceeded in this examination for the following models: 

- cs repetitions, 

- neo-election, 

- planning. 

These models constitute challenges for the next edition. 

5.3 Radars by models 

Figure 1 represents graphically through a set of radar diagrams the highest parameter reached by the 
tools, for each model. Each diagram corresponds to one model, e.g., echo or Kanban. Each diagram is 
divided in ten slices, one for each competing tool, always at the same position. 

The length of the slice corresponds to the highest parameter reached by the tool. When a slice does 
not appear, the tool could not process even the smallest parameter. For instance, AlPiNA handles some 
parameters for eratosthenes and FMS, but is not able to handle the smallest parameter for MAPK. The 
figure also shows that Ma rcie handles all parameters for eratosthenes and FMS, but does not reach the 
highest one for lamport fmea. 

Note that the scale depends on the model : when the parameters of a model vary within a small range 
(less than 100), a linear scale is used (showed using loosely dashed circles as in the Peterson model), 
whereas a logarithmic scale is used for larger parameter values (showed using densely dashed circles 
as in the Philosophers model). We also show dotted circles for the results of the tools, in order to allow 
easier comparison. 

Some models (rwmutex and echo) have complex parameters, built from two values. Tools were only 
able to handle variation of the second parameter, so we only represent it in the figure, in order to show 
an integer value. 

5.4 Radars by tools 

Figure 2 presents also radar diagrams showing graphically the participation and results by tool. Each 
slice in the radars represents a model, always at the same position. 

These diagrams differ from those in Figure 1. Each diagram contains two circles: a slice reaches the 
inner circle if the tool participates to the state space competition for this model, but fails. The size of the 
slice between the inner and the outer circles represents how many parameters are handled. Its length is 
the ratio between the number of handled parameters (without failure) and the total number of parame- 
ters. Thus, we do not take into account the parameter values. 

Two interpretations can rise from the observation of these diagrams: 
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- First, the colored surface in the inner circle shows how many models could be read by the tool. 
For instance, AlPiNA, ITS-Tools and Marcie seem to be non-specialized tools, contrary to 
Crocodile or Neco. This interpretation is biased, as a small surface can also mean that the tool 
developer did not have time to handle all models. 

- Second, the surface between the inner and outer circles shows how well the tool can handle a model. 
Here, we see that ITS-Tools is very efficient for five models, where it handles all parameters. 



Note that LoLa-binstore, LoLa-bloom and Sa ra did not participate to the state space examina- 
tion. 

5.5 Charts for Models 

The charts below correspond to the way tools cope with state space generation for the model that 
were supported. 
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Charts for Peterson 
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Charts for 
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Charts for TokenRing 
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5.6 Execution Charts for AlPiNA 

We provide here the execution charts observed for AlPiNA over the models it could compete with. 

Executions for cs_repetitions 1 chart has been generated. 

AlPiNa, CPU/Memory over execution (or cs-repetitions (25) 
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Executions for echo 1 chart has been generated. 

AlPiNa, CPU/Memory over execution for echo (d2r1 1) 
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Executions for eratosthenes 7 charts have been generated. 
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AlPiNa, CPU/Memory over execution for eratosthenes (020) 



AlPiNa, CPU/Memory over execution for eratosthenes (050) 
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Executions for FMS 5 charts have been generated. 

AlPiNa, CPU/Memory over execution for FMS (2) 
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AlPiNa, CPU/Memory over execution for FMS (50) 
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Executions for galloc_res 1 chart has been generated. 

AlPiNa, CPU/Memory over execution for galloc-res (3} 
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Executions for Kanban 4 charts have been generated. 

AlPiNa, CPU/Memory over execution for Kanban (5) 
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AlPiNa, CPU/Memory over execution for Kanban (20) 
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Executions for lamport_fmea 3 charts have been generated. 
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AlPiNa, CPU/Memory over execution for lamport-fmea (2) 
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Executions for MAPK 1 chart has been generated. 

AlPiNa, CPU/Memory over execution for MAPK (8) 
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Executions for neo-election 1 chart has been generated. 
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Executions for Peterson 3 charts have been generated. 

AlPiNa, CPU/Memory over execution for Peterson (2) 
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Executions for philo_dyn 2 charts have been generated. 

AlPiNa, CPU/Memory over execution for philo-dyn (2) 



100% 
80% 
60% 
40% 
20% 
0% 



100% 
80% 
60% 
40% 
20% 
0% 



# *• <5> & % 

execution time (s) 
CPU Memory 

AlPiNa, CPU/Memory over execution for philo-dyn (10) 
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AlPiNa, CPU/Memory over execution for philo-dyn (3) 
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Executions for Philosophers 8 charts have been generated. 

AlPiNa, CPU/Memory over execution (or Philosophers (5) 
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AlPiNa, CPU/Memory over execution for Philosophers (100) 
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Executions for planning 1 chart has been generated. 



AlPiNa, CPU/Memory over execution for planning (fixed) 
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Executions for railroad 2 charts have been generated. 



AlPiNa, CPU/Memory over execution for railroad (005) 



AlPiNa, CPU/Memory over execution for railroad (010) 
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Executions for ring 1 chart has been generated. 



AlPiNa, CPU/Memory over execution for ring (fixed) 
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Executions for rw_mutex 5 charts have been generated. 



AlPiNa, CPU/Memory over execution for rwmutex (r10w10) 



AlPiNa, CPU/Memory over execution for rwmutex (r10w20) 
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AtPiNa, CPU/Memory over execution for rwmutex (r10w50) 



AlPiNa, CPU/Memory over execution for rwmutex (r1 0w1 00) 
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AlPiNa, CPU/Memory over execution for rwmutex (r10w500) 
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Executions for simplejbs 2 charts have been generated. 



AlPiNa, CPU/Memory over execution for simple-lbs (2} 



AlPiNa, CPU/Memory over execution for simple-lbs (5) 
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Executions for SharedMemory 4 charts have been generated. 



AlPiNa, CPU/Memory over execution for SharedMemory (5) 



AlPiNa, CPU/Memory over execution for SharedMemory (10) 
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AlPiNa, CPU/Memory over execution for SharedMemory (20) 



AlPiNa, CPU/Memory over execution for SharedMemory (50) 
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Executions for ToKenRing 2 charts have been generated. 

AlPiNa, CPU/Memory over execution for TokenRing (5) 
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AlPiNa, CPU/Memory over execution for TokenRing (10) 
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5.7 Execution Charts for Crocodile 

We provide here the execution charts observed for Crocodile over the models it could compete with. 

Executions for cs_repetitions 1 chart has been generated. 

Crocodile, CPU/Memory over execution for cs-repetitions (25) 
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Executions for galloc_res 2 charts have been generated. 

Crocodile, CPU/Memory over execution for galloc-res (3) 
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Crocodile, CPU/Memory over execution for galloc-res (5) 
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Executions for SharedMemory 6 charts have been generated. 



Crocodile, CPU/Memory over execution for SharedMemory (5) 
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Crocodile, CPU/Memory over execution for SharedMemory {1 00) 



<"& "o "o 
execution time fs) 
CPU Memory 



Crocodile, CPU/Memory over execution for SharedMemory (10) 



100% — 
80% - 
60% - 
40% - 
20% - 
0% — 



/ i? ■? $ S > 

execution time (s) 

CPU Memory 
Crocodile, CPU/Memory over execution for SharedMemory (50) 



100% 
80% - 
60% 
40% - 
20% ~ 
0% — 



execution time (s) 
CPU Memory — 

Crocodile, CPU/Memory over execution for SharedMemory (200) 



100% r- 
80% - 
60% i 
40% I- 
20% - 
0% — 



execution time (s) 
CPU Memory 



5.8 Execution Charts for Helena 

We provide here the execution charts observed for Helena over the models it could compete with. 

Executions for lamport_fmea 1 chart has been generated. 

Helena, CPU/Memory over execution for lamport-fmea (2) 
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Executions for neo-election 1 chart has been generated. 
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Executions for philo_dyn 6 charts have been generated. 

Helena, CPU/Memory over execution for philo-dyn (2) 



100% 
80% 
60% 
40% 
20% 
0% 



100% 
80% 
60% 
40% 
20% 
0% 



100% 
80% 
60% 
40% 
20% 
0% 



execution time (s) 
CPU Memory — 

Helena, CPU/Memory over execution for philo-dyn (10) 
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Helena, CPU/Memory over execution for philo-dyn (50) 
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Executions for Philosophers 4 charts have been generated. 
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Executions for SharedMemory 4 charts have been generated. 



Helena, CPU/Memory over execution for SharedMemory (5) 
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Helena, CPU/Memory over execution for SharedMemory (10) 



100% i— 
80% 
60% - 
40% 
20% 
0% — 



100% 



s 'o <s- *%> *%• *0 %■ 

execution time (s) 

CPU Memory 
Helena, CPU/Memory over execution for SharedMemory (50) 



80% - 
60% - 
40% - 

20% ~_ 

0% — 
/ 



i? v S 

execution time (s) 

CPU Memory 



31 



Raw Report on the Model Checking Contest @ Petri Nets 2012 



F. Kordon, A. Linard et. al. 



Executions for TokenRing 4 charts have been generated. 
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5.9 Execution Charts for ITS-Tools 

We provide here the execution charts observed for ITS-Tools over the models it could compete with. 

Executions for echo 1 chart has been generated. 



ITS-Tools, CPU/Memory over execution for echo (d2r1 1) 
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Executions for eratosthenes 7 charts have been generated. 



ITS-Tools, CPU/Memory over execution for eratosthenes (005) 
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ITS-Tools, CPU/Memory over execution for eratosthenes (010) 
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ITS-Tools, CPU/Memory over execution for eratosthenes (020) 
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Executions for FMS 7 charts have been generated. 

ITS-Tools, CPU/Memory over execution for FMS (2) 
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ITS-Tools, CPU/Memory over execution for FMS (10) 
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ITS-Tools, CPU/Memory over execution for FMS (50) 



ITS-Tools, CPU/Memory over execution for FMS (100) 
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ITS-Tools, CPU/Memory over execution for FMS (200) 
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Executions for galloc_res 2 charts have been generated. 



ITS-Tools, CPU/Memory over execution for galloc-res (3) 



ITS-Tools, CPU/Memory over execution for galloc-res (5) 
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Executions for Kanban 8 charts have been generated. 



ITS-Tools, CPU/Memory over execution for Kanban (5) 
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ITS-Tools, CPU/Memory over execution for Kanban (20) 



ITS-Tools, CPU/Memory over execution for Kanban (50) 
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Executions for lamport_fmea 5 charts have been generated. 



ITS-Tools, CPU/Memory over execution for lamport-fmea (2) 
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ITS-Tools, CPU/Memory over execution for lamport-fmea (4) 
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ITS-Tools, CPU/Memory over execution for lamport-fmea (3) 
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ITS-Tools, CPU/Memory over execution for lamport-fmea (5) 



execution time (s) 
CPU Memory 



35 



Raw Report on the Model Checking Contest @ Petri Nets 2012 



F. Kordon, A. Linard et. al. 



ITS-Tools, CPU/Memory over execution for lamport-fmea (6) 
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Executions for MAPK 5 charts have been generated. 

ITS-Tools, CPU/Memory over execution for MAPK (8) 
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ITS-Tools, CPU/Memory over execution for MAPK (40) 
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Executions for neo-election 1 chart has been generated. 



ITS-Tools, CPU/Memory over execution for neo-election (2) 
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Executions for philo_dyn 6 charts have been generated. 



ITS-Tools, CPU/Memory over execution for philo-dyn (2) 



ITS-Tools, CPU/Memory over execution for philo-dyn (3) 
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Executions for Philosophers 13 charts have been generated. 



ITS-Tools, CPU/Memory over execution for Philosophers (5) 
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ITS-Tools, CPU/Memory over execution for Philosophers (20) 



ITS-Tools, CPU/Memory over execution for Philosophers (50) 
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ITS-Tools, CPU/Memory over execution for Philosophers (100) 
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ITS-Tools, CPU/Memory over execution for Philosophers (500) 
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ITS-Tools, CPU/Memory over execution for Philosophers (1 000) 
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ITS-Tools, CPU/Memory over execution for Philosophers (2000) 



execution time (s) 
CPU Memory 



ITS-Tools, CPU/Memory over execution for Philosophers (5000) 

100% | ; : : : 1 

80% - 
60% - 
40% 
20% - 
0% 

/ <? v & S 

execution time (s) 

CPU Memory — 



ITS-Tools, CPU/Memory over execution for Philosophers (10000) 



ITS-Tools, CPU/Memory over execution for Philosophers (50000) 



<? v & 

execution time (s) 

CPU Memory — 



100% 
80% 
60% 
40% 
20% 
0% 



^ i? v 

execution time (s) 

CPU Memory 



38 



Raw Report on the Model Checking Contest @ Petri Nets 2012 



F. Kordon, A. Linard et. al. 



ITS-Tools, CPU/Memory over execution for Philosophers {1 00000) 
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Executions for planning 1 chart has been generated. 



ITS-Tools, CPU/Memory over execution for planning (fixed) 
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Executions for ring 1 chart has been generated. 

ITS-Tools, CPU/Memory over execution for ring (fixed) 
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Executions for rw_mutex 4 charts have been generated. 



ITS-Tools, CPU/Memory over execution for rwmutex (r10w10) 
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ITS-Tools, CPU/Memory over execution for rwmutex (r10w50) 
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ITS-Tools, CPU/Memory over execution for rwmutex (r10w100) 
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Executions for SharedMemory 4 charts have been generated. 



ITS-Tools, CPU/Memory over execution for SharedMemory (5) 



ITS-Tools, CPU/Memory over execution for SharedMemory (10) 
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ITS-Tools, CPU/Memory over execution for SharedMemory (20) 



ITS-Tools, CPU/Memory over execution for SharedMemory (50) 
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Executions for simplejbs 5 charts have been generated. 



ITS-Tools, CPU/Memory over execution for simple-lbs (2) 



ITS-Tools, CPU/Memory over execution for simple-lbs (5) 
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ITS-Tools, CPU/Memory over execution for simple-lbs (10) 
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ITS-Tools, CPU/Memory over execution for simple-lbs (20) 
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Executions for TokenRing 4 charts have been generated. 



ITS-Tools, CPU/Memory over execution for TokenRing (5) 



ITS-Tools, CPU/Memory over execution for TokenRing (1 0} 
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ITS-Tools, CPU/Memory over execution for TokenRing (20) 
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ITS-Tools, CPU/Memory over execution for TokenRing (50) 
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5.10 Execution Charts for Marcie 

We provide here the execution charts observed for Marcie over the models it could compete with. 

Executions for cs_repetitions 1 chart has been generated. 

Marcie, CPU/Memory over execution for cs-repetitions (25) 
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Executions for echo 2 charts have been generated. 



Marcie, CPU/Memory over execution for echo (d2r1 1 ) 



Marcie, CPU/Memory over execution for echo (d2r13) 
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Executions for eratosthenes 7 charts have been generated. 



Marcie, CPU/Memory over execution for eratosthenes (005) 



Marcie, CPU/Memory over execution for eratosthenes (01 0) 
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Marcie, CPU/Memory over execution for eratosthenes (020) 



Marcie, CPU/Memory over execution for eratosthenes (050) 
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Marcie, CPU/Memory over execution for eratosthenes (1 00) 



Marcie, CPU/Memory over execution for eratosthenes (200) 
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Marcie, CPU/Memory over execution for eratosthenes (500) 



100% 
80% 
60% 
40% 
20% 
0% 



<s- *b °o 

execution time (s) 
CPU Memory 



Executions for FMS 7 charts have been generated. 

Marcie, CPU/Memory over execution for FMS (2) 
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Marcie, CPU/Memory over execution for FMS (10) 
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Marcie, CPU/Memory over execution for FMS (50) 
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Marcie, CPU/Memory over execution for FMS (200) 
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Marcie, CPU/Memory over execution for FMS (5} 
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Executions for galloc_res 2 charts have been generated. 

Marcie, CPU/Memory over execution for galloc-res (3) 
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Marcie, CPU/Memory over execution for galloc-res (5) 
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Executions for Kanban 6 charts have been generated. 

Marcie, CPU/Memory over execution for Kanban (5) 
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Marcie, CPU/Memory over execution for Kanban (20} 



execution time (s) 
CPU Memory 

Marcie, CPU/Memory over execution for Kanban (1 00) 
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Marcie, CPU/Memory over execution for Kanban (10) 
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Marcie, CPU/Memory over execution for Kanban (50) 
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Executions for lamport_fmea 3 charts have been generated. 
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Marcie, CPU/Memory over execution for lamport-fmea (2) 
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Marcie, CPU/Memory over execution for lamport-fmea (4) 
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Marcie, CPU/Memory over execution for lamport-fmea (3) 
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Executions for neo-election 1 chart has been generated. 
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Marcie, CPU/Memory over execution for neo-election (2) 
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Executions for Peterson 2 charts have been generated. 

Marcie, CPU/Memory over execution for Peterson (2} 
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Marcie, CPU/Memory over execution for Peterson (3) 
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Executions for philo_dyn 3 charts have been generated. 



Marcie, CPU/Memory over execution for philo-dyn (2) 



Marcie, CPU/Memory over execution for philo-dyn (3) 
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Marcie, CPU/Memory over execution for philo-dyn (10) 
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Executions for Philosophers 10 charts have been generated. 



Marcie, CPU/Memory over execution for Philosophers (5) 



Marcie, CPU/Memory over execution for Philosophers (10) 
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Marcie, CPU/Memory over execution for Philosophers (20) 



Marcie, CPU/Memory over execution for Philosophers (50) 
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Marcie, CPU/Memory over execution for Philosophers (1 00) 



Marcie, CPU/Memory over execution for Philosophers (200} 
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Marcie, CPU/Memory over execution for Philosophers (2000) 
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Marcie, CPU/Memory over execution for Philosophers (1000) 
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Marcie, CPU/Memory over execution for Philosophers (5000) 
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Executions for railroad 3 charts have been generated. 
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Marcie, CPU/Memory over execution for railroad (020) 
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Marcie, CPU/Memory over execution for railroad (01 0) 
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Executions for ring 1 chart has been generated. 

Marcie, CPU/Memory over execution for ring (fixed) 
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Executions for rw_mutex 5 charts have been generated. 

Marcie, CPU/Memory over execution for rwmutex (r10w10) 
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Marcie, CPU/Memory over execution for rwmutex (r10w50) 
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Marcie, CPU/Memory over execution for rwmutex (r10w20) 
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Marcie, CPU/Memory over execution for rwmutex (r10w100) 
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Executions for SharedMemory 7 charts have been generated. 



Marcie, CPU/Memory over execution for SharedMemory (5) 
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Marcie, CPU/Memory over execution for SharedMemory (20) 



execution time (s) 
CPU Memory — 

Marcie, CPU/Memory over execution for SharedMemory {1 00) 
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Marcie, CPU/Memory over execution for SharedMemory (500) 



Marcie, CPU/Memory over execution for SharedMemory (10) 
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Marcie, CPU/Memory over execution for SharedMemory (50) 
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Marcie, CPU/Memory over execution for SharedMemory (200) 
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Executions for simplejbs 3 charts have been generated. 
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Marcie, CPU/Memory over execution for simple-lbs (2) 
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Marcie, CPU/Memory over execution for simple-lbs (10) 
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Marcie, CPU/Memory over execution for simple-lbs (5) 
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Executions for TokenRing 4 charts have been generated. 



100% 

80% 
60% 
40% 
20% 
0% 



100% 
80% 
60% 
40% 
20% 
0% 



Marcie, CPU/Memory over execution for TokenRing (5) 
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Marcie, CPU/Memory over execution for TokenRing (20} 
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Marcie, CPU/Memory over execution for TokenRing (10) 
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Marcie, CPU/Memory over execution for TokenRing (50) 
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5.11 Execution Charts for Neco 

We provide here the execution charts observed for Neco over the models it could compete with. 
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Executions for eratosthenes 4 charts have been generated. 
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Neco, CPU/Memory over execution for eratosthenes (020) 
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Neco, CPU/Memory over execution for eratosthenes (050) 
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Executions for Kanban 2 charts have been generated. 

Neco, CPU/Memory over execution for Kanban (5) 



100% 
80% 
60% 
40% 
20% 
0% 



's *o 

execution time (s) 
CPU Memory 



100% r- 
80% 
60% 
40% - 
20% 

0% L 



Neco, CPU/Memory over execution for Kanban (10) 
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Executions for MAPK 2 charts have been generated. 

Neco, CPU/Memory over execution for MAPK (8) 
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Neco, CPU/Memory over execution for MAPK (20) 
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Executions for Peterson 2 charts have been generated. 



Neco, CPU/Memory over execution for Peterson (2) 



Neco, CPU/Memory over execution for Peterson (3) 
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Executions for philo_dyn 3 charts have been generated. 



Neco, CPU/Memory over execution for philo-dyn (2) 



Neco, CPU/Memory over execution for philo-dyn (3) 
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Neco, CPU/Memory over execution for philo-dyn (10) 
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Executions for rw_mutex 5 charts have been generated. 



Neco, CPU/Memory over execution for rwmutex (r10w10) 



Neco, CPU/Memory over execution for rwmutex (r10w20) 
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Neco, CPU/Memory over execution for rwmutex (r10w50) 



Neco, CPU/Memory over execution for rwmutex (r10w100) 
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Neco, CPU/Memory over execution for rwmutex (r10w500) 
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Executions for simplejbs 3 charts have been generated. 

Neco, CPU/Memory over execution for simple-lbs (2) 
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Neco, CPU/Memory over execution for simple-lbs (5) 
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5.12 Execution Charts for PNXDD 

We provide here the execution charts observed for PNXDD over the models it could compete with. 
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Executions for cs_repetitions 1 chart has been generated. 

PNXDD, CPU/Memory over execution for cs-repetitions (25) 
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Executions for FMS 6 charts have been generated. 

PNXDD, CPU/Memory over execution for FMS (2) 
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PNXDD, CPU/Memory over execution for FMS (10) 



^> i? v $ <r > 
execution time (s) 

CPU Memory 
PNXDD, CPU/Memory over execution for FMS (50) 



•?o 3 o "*o & 
execution time (s) 

CPU Memory 



100% r- 
80% - 
60% - 
40% - 
20% - 
0% - 



100% 
80% 
60% 



40% 
20% 

0% — 



100% 
80% 
60% 
40% 
20% 
0% 



PNXDD, CPU/Memory over execution for FMS (5) 



^> i? v & & > 

execution time (s) 

CPU Memory — 

PNXDD, CPU/Memory over execution for FMS (20) 



execution time (s) 
CPU Memory 

PNXDD, CPU/Memory over execution for FMS (100) 
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Executions for galloc_res 2 charts have been generated. 

PNXDD, CPU/Memory over execution for galloc-res (3) 
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PNXDD, CPU/Memory over execution for galloc-res (5) 
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Executions for Kanban 5 charts have been generated. 

PNXDD, CPU/Memory over execution for Kanban (5) 
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PNXDD, CPU/Memory over execution for Kanban (20) 



execution time (s) 
CPU Memory 

PNXDD, CPU/Memory over execution for Kanban (100) 
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Executions for MAPK 5 charts have been generated. 



PNXDD, CPU/Memory over execution for MAPK (8) 



PNXDD, CPU/Memory over execution for MAPK (20) 
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PNXDD, CPU/Memory over execution for MAPK (40) 



PNXDD, CPU/Memory over execution for MAPK (80) 
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PNXDD, CPU/Memory over execution for MAPK (160) 
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Executions for Peterson 4 charts have been generated. 



PNXDD, CPU/Memory over execution for Peterson (2) 



PNXDD, CPU/Memory over execution for Peterson (3) 
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PNXDD, CPU/Memory over execution for Peterson (4) 



PNXDD, CPU/Memory over execution for Peterson (5) 
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Executions for philo_dyn 3 charts have been generated. 



PNXDD, CPU/Memory over execution for philo-dyn (2) PNXDD, CPU/Memory over execution for philo-dyn (3) 
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Executions for Philosophers 6 charts have been generated. 



PNXDD, CPU/Memory over execution for Philosophers (5) 
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PNXDD, CPU/Memory over execution for Philosophers (20) 



PNXDD, CPU/Memory over execution for Philosophers (50) 
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PNXDD, CPU/Memory over execution for Philosophers (100) 
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PNXDD, CPU/Memory over execution for Philosophers (200) 
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Executions for SharedMemory 4 charts have been generated. 



PNXDD, CPU/Memory over execution for SharedMemory (5) 
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PNXDD, CPU/Memory over execution for SharedMemory (20) 
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PNXDD, CPU/Memory over execution for SharedMemory (10) 
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Executions for simplejbs 5 charts have been generated. 



PNXDD, CPU/Memory over execution for simple-lbs (2) 



PNXDD, CPU/Memory over execution for simple-lbs (5) 
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PNXDD, CPU/Memory over execution for simple-lbs (10) 



PNXDD, CPU/Memory over execution for simple-lbs (15) 
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PNXDD, CPU/Memory over execution for simple-lbs (20) 
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Executions for TokenRing 3 charts have been generated. 



PNXDD, CPU/Memory over execution for TokenRing (5) 



PNXDD, CPU/Memory over execution for TokenRing (10) 
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PNXDD, CPU/Memory over execution for TokenRing (20) 
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6 Raw Data for Structural Formulae Evaluation 

This section shows the raw results of the structural formulae examination. Table 4 summarizes the 
highest scaling parameter reached by the tools for each model. Then Charts generated from the data 
collected for this examination are provided. This table should be interpreted using the legend displayed 
in Table 2, page 11. Let us not that only two tools did participate in the evaluation of structural formulas: 
AlPiNA and Helena. LoLa-binstore, LoLa-bloom and Sara could not compete because of the com- 
bination of structural assertions. 
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Table 4. Results for the structural formulae examination 



Section 6.1 presents the data computed by tools. Then, Section 6.2 shows how models have been 
handled by tools and Sections 6.3 and 6.4 summarize how tools did cope with models. 

Section 6. 1 shows that no tool computes the same set of formulae. This is why we do not display the 
charts that have been extracted from the executions since they have no meaning at all. 

6. 1 Computed results for the formulae 

Since only two tools did compete and had sometimes different results, we could not consolidate the 
values and decided to show the results as they were produced in Table 5. For each scaling parameter, 
we proposed a set of formulae. Outputs from the tools were analyzed and formula were sorted by their 
identifier in order to build a vector where the i th element corresponds to the i th formula of the set 
(value F or T). Sometimes, the tool cannot compute a formula. We then display a "." instead. For this 
examination, it appears that, the problem is often due to formula evaluation itself (the grammar was 
published late and thus). "?" means that the tool participated but did not compute any result. 

6.2 Processed Models 

This section summarizes how models were processed by tools. Let us first note that no tool suc- 
ceeded in this examination for the following models: 

- cs repetitions, 
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Table 5. Results of structural formulae evaluation for the models where at least one tool competed 
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- echo, 

- eratosthenes, 

- gallocres, 

- MAPK, 

- neo-election, 

- planning, 

- railroad, 

- ring. 

These models constitute challenges for the next edition of the Model Checking Contest. 

6.3 Radars by models 

Figure 3 represents graphically through a set of radar diagrams the highest parameter reached by the 
tools, for each model. Each diagram corresponds to one model, e.g., echo or Kanban. Each diagram is 
divided in ten slices, one for each competing tool, always at the same position. 

The length of the slice corresponds to the highest parameter reached by the tool. When a slice does 
not appear, the tool could not process even the smallest parameter. For instance, Helena handles some 
parameters for lamport fmea and TokenRing, but is not able to handle the smallest parameter for er- 
atosthenes. The figure also shows that only two tools (AlPiNA and Helena) did compete for this exami- 
nation. None of them achieves the highest parameter for any model. 

Note that the scale depends on the model : when the parameters of a model vary within a small range 
(less than 100), a linear scale is used (showed using loosely dashed circles as in the Peterson model), 
whereas a logarithmic scale is used for larger parameter values (showed using densely dashed circles 
as in the Philosophers model). We also show dotted circles for the results of the tools, in order to allow 
easier comparison. 

Some models (rwmutex and echo) have complex parameters, built from two values. Tools were only 
able to handle variation of the second parameter, so we only represent it in the figure, in order to show 
an integer value. 

6.4 Radars by tools 

Figure 4 presents also radar diagrams showing graphically the participation and results by tool. Each 
slice in the radars represents a model, always at the same position. 

These diagrams differ from both those in Figure 3 and those in Figure 1. Each diagram contains two 
circles, as in Figure 1: a slice reaches the inner circle if the tool participates to the state space compe- 
tition for this model, but fails. The number of subslices between the inner and the outer circles repre- 
sents how many parameters are handled. The angle covered by each subslice shows the ratio between 
the computed formulas and the total number of formula? in the examination. For instance, AlPiNA has 
wider subslices than Helena: it is able to handle more formulae. But its subslices do not cover the whole 
angle dedicated to each model: AlPiNA could not handle all the formulas proposed in this examination. 

The colored surface in the inner circle shows if the tool could at least handle one formula for the 
model. For instance, AlPiNA and Helena could handle formulas for simple lbs, but not for railroad. 

This figure is not sufficient as we do not clearly distinguish tools that try to handle formulas from 
tools that do not compete. For instance, AlPiNA tries for all models, but has four "blank" models in the 
figure. 
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Fig. 3. Highest parameter reached for each model, in structural formulae evaluation 
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Fig. 4. Handled parameters for each tool, in structural formulas evaluation 
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7 Raw Data for Reachability Formulae Evaluation 

This section shows the raw results of the reachability formulas examination. Table 6 summarizes the 
highest scaling parameter reached by the tools for each model. Then Charts generated from the data 
collected for this examination are provided. This table should be interpreted using the legend displayed 
in page 11. 
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Table 6. Results for the reachability formulae examination 



Section 7.1 presents the data computed by tools. Then, Section 7.2 shows how models have been 
handled by tools and Section 7.3 summarizes how tools did cope with models. 

Section 7.1 shows that no tool compute the same set of formulae. This is why we do not display the 
charts that have been extracted from the executions since they have no meaning at all. 

7. 1 Computed results for the formulae 

Since the participating tools did produce various values, we could not consolidate the results and 
decided to show them as they were produced in Tables 7 and 8. We use the notations already presented 
for Table 5, page 63 (use of T, F or "." in the issued vector). 

7.2 Processed Models 

This section summarizes how models were processed by tools. Let us first note that no tool suc- 
ceeded in this examination for the following models: 

- planning, 

- railroad, 

- ring. 



These models constitute challenges for the next edition of the Model Checking Contest. 
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Table 7. Results of reachability formulae evaluation for the models where at least one tool produced a 
result 
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Table 8. Results of reachability formulae evaluation for the models where at least one tool produced a 
result (continued) 
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7.3 Radars by models 

Figure 5 represents graphically through a set of radar diagrams the highest parameter reached by the 
tools, for each model. Each diagram corresponds to one model, e.g., echo or Kanban. Each diagram is 
divided in ten slices, one for each competing tool, always at the same position. 

The length of the slice corresponds to the highest parameter reached by the tool. When a slice does 
not appear, the tool could not process even the smallest parameter. For instance, LoLa-binsto re han- 
dles some parameters for almost all models (and the highest parameter for most of them), but is not able 
to handle the smallest parameter for Shared Memory. 

Note that the scale depends on the model : when the parameters of a model vary within a small range 
(less than 100), a linear scale is used (showed using loosely dashed circles as in the Peterson model), 
whereas a logarithmic scale is used for larger parameter values (showed using densely dashed circles 
as in the Philosophers model). We also show dotted circles for the results of the tools, in order to allow 
easier comparison. 

Some models (rwmutex and echo) have complex parameters, built from two values. Tools were only 
able to handle variation of the second parameter, so we only represent it in the figure, in order to show 
an integer value. 

7.4 Radars by tools 

Figure 6 presents also radar diagrams showing graphically the participation and results by tool. Each 
slice in the radars represents a model, always at the same position. 

These diagrams differ from both those in Figure 5 and those in Figure 1. Each diagram contains two 
circles, as in Figure 1 : a slice reaches the inner circle if the tool participates to the state space competition 
for this model, but fails. The number of subslices between the inner and the outer circles represents 
how many parameters are handled. The angle covered by each subslice shows the ratio between the 
computed formulas and the total number of formulas in the examination. For instance, LoLa-binsto re 
has wider subslices than LoLa-bloom: it is able to handle more formulas. But its subslices do not cover 
the whole angle dedicated to each model: LoLa-binsto re could not handle all the formulas proposed 
in this examination. For eratosthenes, Sa ra has a very wide angle, so it handles far more formulas than 
for the other models. 

The colored surface in the inner circle shows if the tool could at least handle one formula for the 
model. For instance, AlPiNA and Helena could handle formulas for simple lbs, but not for railroad. 

This figure is not sufficient as we do not clearly distinguish tools that try to handle formulas from 
tools that do not compete. For instance, AlPiNA tries for all models, but has four "blank" models in the 
figure. 
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8 RowData for CTL and LTL Formulae Evaluation 

No tool participated in these two examinations. 
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9 Conclusion 

This paper reported our experience with the second Model Checking Contest @ Petri nets 2012. 

From the tool developers' point of view, such an event allows to compare tools on a common bench- 
mark that could become a public repository. Also, some mechanisms established for the contest, such 
as a language to elaborate the formula to be verified could become, over the years, a common way to 
provide formulae to the various tools developed by the community. 

Some difficulties in the generation of formulae for the contest led to a very late publication of the 
grammar. Moreover, it appeared that some difficulties were underestimated for tool integration. This 
explain why, while the analysis of the state space generation examination is fruitful, this is not the case 
for the formulae evaluation ones. 

Thus, we provide in this report raw results as they were computed, and with no interpretation, in 
order to emphasize the effort performed by both the organizers and the tool submitters. 

Lesson Learned from MCC'2012 This edition was greatly improved from the lesson learned during 
the previous edition (see [29]). However, some remarks about the complexity of the submission proce- 
dure led to the following suggestions: 

i structural formulae examination will be split in two subcategories: a single request one (asking for 
deadlock, bound, etc.) and a combined one where these single requests can be combined. 

ii for formulaae examinations, we would like to distinguish satisfied properties from unsatisfied ones. 
This would provide a better feedback for tool developers (due to the difficulties encountered this 
year, it was impossible to complete this for the 2012 edition). 

iii the structure of a model x scaling value will be simplified, thus avoiding the complex naming prob- 
lems we encountered this year (especially for fomulae) . This should simplify the integration of model 
checkers in the image disk. 

iv we will provide, as for last year, a pre-installed disk image with a dummy model checker, just to let 
tool developers see how their tool can be integrated. 

v the "surprise model" examination will be processed as well, it is of interest to evaluate the capability 
of tools with their default settings. We had no time to do so this year. 

Finally, we would like to set-up an on-line repository that would help tool developers to submit more 
models as well as to perform tests on their tools. However, this task requiring more manpower, we do 
not know if it will be operated for MCC'2013. 
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